Áååë Ì Blockinòò Blockin Blockinð Êêôóöø ¾¼¼¾¹¿¾ Ùùù×ø ¾¼¼¾

A lass of odes is said to rea h apa ity C of the binary symmetri hannel if for any rate R < C and any " > 0 there is a suÆ iently large N su h that odes of length N and rate R from this lass provide error probability of de oding at most ", under some de oding algorithm. The study of the error probability of expander odes was initiated in Barg and Z emor (2002), where it was shown that they attain apa ity of the binary symmetri hannel under a linear-time iterative de oding with error probability falling exponentially with ode length N . In this work we study variations on the expander ode onstru tion and fo us on the most important region of ode rates, lose to the hannel apa ity. For this region we estimate the de rease rate (the error exponent) of error probability of de oding for randomized ensembles odes. The resulting estimate gives a substantial improvement of previous results for expander odes and some other expli it ode families. 1 Introdu tion We study transmission of information with linear odes over the binary symmetri hannel (BSC). An [N;K℄ binary linear ode C is a K-dimensional linear subspa e of f0; 1gN : The number N is alled the length of the ode and the relative dimension K=N is alled the rate of the ode, denoted by R = R(C): The BSC is a random f0; 1g ! f0; 1g map a ting independently on the oordinates of the transmitted ode ve tor x: A binary digit sent over the hannel is re eived orre tly with probability 1 p and ipped with probability p < 1=2: The obje tive of the de oder is to restore orre tly the transmitted odeword. Maximum likelihood (or omplete) de oding of C onsists of hoosing a odeword x0 losest to the re eived ve tor y. The event that x 6= x0 orresponds to a de oding error. The probability Pe(C) that de oding goes wrong is independent of the transmitted odeword and is a polynomial fun tion of p. This polynomial is notoriously diÆ ult to ompute exa tly but estimating the value Pe(C) an be somewhat simpli ed: when p is small enough Pe(C) behaves like its lowest-degree term and the lowest degree equals half the minimum Hamming distan e of C. For this reason, ombinatorial oding theory is on erned with the onstru tion of large odes with large minimum distan e. However, for both theoreti al and pra ti al reasons (like the emergen e of mobile ommuni ations and their very noisy hannels), there has been a renewed interest for studying the situation when p is large, or equivalently when R is lose to apa ity. Shannon's theorem states that Pe(C) an stay lose to zero only as long as we have R < C where C is the hannel apa ity and depends only on p. Furthermore, when R is lose to C, we know that for large N , xed rate R < C, and for the best possible odes of rate R, the de oding error probability Pe(C) takes the form Pe(C) = 2 E(R;p)N+o(N) where E(R; p), alled the error exponent, is a positive quantity that depends only on p and the rate R and an be omputed exa tly [7℄. Unfortunately, the only known way to a hieve this best possible error exponent is to use a random ode C together with de oding algorithms that nd the losest odeword by essentially performing exhaustive sear h over C. Turning to manageable de oding algorithms, until fairly re ently only one lass of odes was known to a hieve a non-zero de oding exponent in polynomial time (though less than E(R; p)) for rates arbitrarily lose to hannel apa ity: the lass of on atenated odes, introdu ed by Forney [8℄ and extensively studied through the mid-eighties (see [1℄, [6℄ for overviews). In the nineties the dis overy of turboodes [3℄ with their largely unexplained lose-toapa ity performan e shifted emphasis to iterative de oding te hniques. One parti ular lass of odes that an be iteratively de oded is that of expander odes. An expander ode is onstru ted by assigning binary digits to edges of a bipartite graph in a way that was introdu ed by Tanner [14℄. A surge of interest in them o urred after it was shown in [13℄ that if the underlying graph is an expander graph, then they orre t an (N) number of errors under an O(N) iterative de oding. Not only was this a signi ant a hievement for iterative de oding but it was the rst example of this kind in oding theory at large; indeed, the on atenated de oding rival requires an O(N2) de oding time. { 2 { In our work [2℄ we employ the iterative de oding algorithm of [15℄ to show that expander odes a tually rea h the apa ity of the BSC with a positive error exponent. Though the error exponent of on atenated odes [8℄ is better than that of expander odes of [2℄, their performan e again relies on a quadrati -time de oding algorithm as opposed to linear-time de oding of [15℄,[2℄. An expander ode C in [13℄, [15℄ is onstru ted from a bipartite -regular graph G = [A [ B;E℄ with both parts of size n and a binary ode [ ; R ℄ ode C0. Coordinates of a odeword in C are in one-to-one orresponden e with edges of G and satisfy the ondition that the subve tor in ident to every vertex v 2 A [ B is a odeve tor in C0. The de oding algorithm of [15℄ onsists of iterations that alternate between A and B. In ea h iteration all the verti es of the respe tive part are de oded in parallel with the ode C0: Paper [15℄ also provides a bound on the number of errors orre table by this de oding. Among the new ideas introdu ed in [2℄ is the use of two di erent odes: C0 for the part A and C1 for B. The value of the resulting error exponent is then optimized on the hoi e of the rates of these odes. Another idea of [2℄ is repla ing ea h edge by t parallel edges and employing both binary and q-ary stru ture in the analysis of omponent odes, q = 2t: In this paper we obtain a substantial improvement of the estimate of the error exponent for expander odes, fo using on R lose to apa ity. In parti ular, we surpass in this region the error exponent of Forney's on atenated odes [8℄, a ben hmark for a long time. The improvement relies on the following ideas whi h were not employed in earlier analysis. In the rst iteration, again relying on the q-ary stru ture of the ode C0, we employ detailed information on the error events. Introdu ing an element of randomization in the overall ode onstru tion, we an then make a stronger statement about performan e of the ode C1 in the se ond de oding iteration. Finally, we modify the original onstru tion by adjoining a number of verti es of degree one to the expander graph. This de reases the error orre ting potential but improves the overall ode rate and the resulting trade-o improves the overall error exponent. This idea borrows from turbo odes for whi h the underlying graph has many degree one verti es ontrary to other lasses of odes amenable to iterative de oding. The rest of the paper is organized as follows. In Se tions 2 and 3 we introdu e the ne essary oding ba kground. In Se tion 4 we summarize previous work on expander odes. In Se tion 5 we introdu e our new variation on de oding and give a re ned probabilisti analysis of its behavior: this results in a rst error exponent in Theorem 7. In Se tion 6 this is improved to Theorem 10 through a modi ed onstru tion that we analyze. Finally we give some on luding omments. 2 Codes and their parameters Although our ultimate goal will be binary odes, we will also onsider odes over larger alphabets of size q = 2t. By Hq = HN q we denote the q-ary Hamming spa e, i.e., the N dimensional oordinate spa e over the eld of q elements. The number of nonzero oordinates of a ve tor x 2 Hq is alled the (Hamming) weight, denoted jxj: The Hamming distan e is de ned by d(x; y) = jx yj: { 3 { For a given linear ode C the minimum weight of a nonzero odeword in C is alled its distan e. For a given q we will use notation C[N;K;D℄ to refer to a linear ode of length N , dimension K and distan e D = D(C); o asionally omitting the distan e. One of the key problems of ombinatorial oding theory is nding the maximum size of a ode C of length N and distan e D. Consider families of odes Ci; i = 1; 2; : : : of growing length Ni, rate Ri and relative distan e Æi = D(Ci)=Ni. A ording to the well known Gilbert-Varshamov (GV) bound there exist sequen es of odes of rate R(Ci) ! R and relative distan e Æi ! Æ for any R < 1 Hq(Æ); where Hq(x) = x logq x q 1 (1 x) logq(1 x) is the q-ary entropy fun tion. For a given R we let Æ(q) GV (R) denote the GV distan e: Æ(q) GV (R) = H 1 q (1 R): Note that Æ(q) GV (0) = (q 1)=q; Æ(q) GV (1) = 0: For q = 2 we omit the supers ript and write simply ÆGV(R): Likewise, throughout the paper if the base of the logarithms and exponents is missing, it is equal to 2. A number of upper bounds are known on the relative distan e Æ(R) of a ode sequen e of rate R. We mention the Bassalygo-Elias bound whi h asserts that, for binary odes Æ(R) ÆE(R) := 2ÆGV (R)(1 ÆGV (R)): We all the quantity ÆE(R) := 2ÆGV(R)(1 ÆGV(R)) the Elias radius; its relevan e to de oding will be ome apparent from Prop. 1. As q = 2t gets large, the quantity (q 1)=q omes lose to one, and the inverse entropy fun tion gets lose to the linear fun tion 1 R: More pre isely, for any ; > 0 there exists a value t0 su h that for all t > t0 and all < R < 1 0 < (1 R) Æ(q) GV (R) < : (1) 3 De oding and error exponents 3.1 De oding Given a ode C Hq, any mapping : Hq ! C that takes a given point y to one of the odewords of C losest to it is alled maximum likelihood de oding. Given an [N;K℄ binary ode C and a BSC(p), we de ne the error probability for a given ode ve tor x as follows: Pe(x) = Pr (y) 6= x j x is sent and y is re eived : { 4 { Be ause C is a linear ode this probability is independent of the hoi e of the mapping and of the ode ve tor x and equals the average error probability Pe(C) = 2 KXx2C Pe(x) Therefore, for the purposes of analysis we always assume that the transmitted ve tor always is the all-zero one: x = 0 and write Pe(C) instead of Pe(0). By the lassi al results of oding theory [7℄, [10℄ there exist sequen es of binary linear odes su h that the probability Pe(C) under maximum likelihood de oding falls exponentially with the ode length N . Therefore, de ne the error exponent E(C) = N 1 logPe(C): We de ne the best attainable error exponent for the rate R as E(R; p) = lim inf N!1 max C HN2 :R(C) RE(C): It was proved in [7℄ (see also [10℄) that E(R) > 0 for 0 R < C: 3.2 Random oding exponent and typi al events The best known existen e bound E0(R; p) on the error exponent E(R; p) of binary odes (linear or not) is obtained by the random oding method [10℄. The fun tion E0(R; p) is positive for all rates below the hannel apa ity C = 1 H(p): It is easy to prove by random hoi e [10℄ that there exist sequen es of binary linear [N;RN ℄ odes that attain the GV bound on the minimum distan e and rea h the error exponent E0(R; p) under maximum likelihood de oding. We assume that p is xed and R varies from zero to C. The form of the bound depends on the lo ation of R with respe t to C: We are interested in the high-rate region, i.e., R lose to C: For high rates we obtain the bound [7℄, [10℄ E0(R; p) = D(ÆGV (R)kp); where D(xky) := x log(x=y) + (1 x) log((1 x)=(1 y)): This bound is a tually tight in the region R rit R C; where the value R rit = 1 H( 0), 0 = pp pp+p1 p; (2) is alled the riti al rate of the hannel. In other words, for R rit R C we have E(R; p) = E0(R; p) [7℄. Note that E0(C; p) = 0: For rates 0 R R rit the random oding exponent has the following form: E0(R; p) = ÆGV(R) log 2pp(1 p) (0 R Rx) E0(R; p) = D( 0kp) +R rit R (Rx R R rit); { 5 { where Rx = 1 H(2 0(1 0)): Both onstru tion and de oding omplexity of known ode sequen es that attain this error exponent grow exponentially with the ode length N . It is possible to say more about the geometry of typi al error events under maximum likelihood de oding of random odes. In the high-rate region whi h interests us, the following proposition is true. Proposition 1 Let p be the parameter of a BSC. Assume that the transmitted ve tor is the all-zero one, and denote by z the de oded ve tor output by maximum likelihood de oding. For any R and for any large enough N , there exists a ode C su h that, for any p su h that R rit R < C, C has error exponent E0(R; p), for any > 0 we have Pr h ÆE(R) jzj N z 6= 0i < 2 N ( ) ( ( ) > 0 is independent of N); where Pr[ ℄ is the probability that the random ve tor z ful lls the ondition in the bra kets. In other words, for high rates the typi al relative distan e between the transmitted odeword and the de oded odeword in the ase of de oding error equals ÆE(R): The proof of this result (see Appendix) relies on a ombination of fa ts known to oding theorists, but not spelled out in the literature. 3.3 Error exponents for O(N2) de oding: Con atenated odes An important example of odes that attain hannel apa ity under lowomplexity de oding is given by Forney's on atenated odes [8℄ (see also [6℄). Con atenated odes form a generalization of Elias's produ t odes and provide ode families with better parameters in terms of both ode distan e and error exponents. Binary [N = nm; k`℄ on atenated odes are onstru ted by rst en oding the k q-ary message symbols, q = 2`; with an [n; k℄ q-ary ode C1, then representing every ode symbol ba k as a string of ` bits, and then en oding it with an [m; `℄ binary ode C0: We assume that both rates R0 = `=m of the inner ode C0 and R1 = k=n of the outer ode are xed. Let m = log2N . As with all linear odes, properties of a typi al on atenated ode found by random hoi e are mu h better than those of the known expli it families. In parti ular, there exists a sequen e of binary [m; ` = mR0℄ odes C0 for whi h the error probability of maximum likelihood de oding falls as 2 mE0(R0;p): Moreover, a brute-for e implementation of de oding of the ode C0 has omplexity O(m2R0m) = O(N logN). An expli it family is obtained by taking as outer odes a sequen e of Reed-Solomon odes C1 of growing length { 6 { n over the alphabet of growing size q = 2`: Performing algebrai (generalized minimum distan e) de oding of the ode C1 with omplexity O(n2); we obtain the error exponent [8℄ EF(R) = max R where = 2 . Let T be the subset of B de ned by T = fv 2 B; dS(v) (1 + ) g: Then we have : jT j  jSj: Proof. The number of edges of the graph GS[T indu ed by S [ T is at least jT j(1 + ) and therefore the average degree dST satis es 2jT j(1 + ) jSj+ jT j : Applying Lemma 2 we therefore have : 2jT j(1 + ) jSj+ jT j 2jSjjT j jSj+ jT j n + when e, 2jT j(1 + ) 2jSjjT j n + (jSj+ jT j) ( =2) 2 jSj and the result follows after some rearranging. 4.2 Expander odes Let G = [A[B;E℄ be as above, with : Let us x an arbitrary order of the edges in E. For a vertex v this de nes an ordering of edges v(1); : : : ; v(Æ) in ident to it. Given a binary ve tor x 2 HN 2 ; this ordering indu es a subve tor xv = (xv(1); : : : ; xv( )): To onstru t a linear ode C asso iated with the graph G we also need two binary odes C0[ ; R0 ; Æ0 ℄ and C1[ ; R1 ; Æ1 ℄: An N -ve tor x is a odeword of C if and only if for every left vertex v the subve tor xv is a odeword in C0 and for every right vertex w, the subve tor xw is a odeword in C1: The ode C has length N = n and rate R R0+R1 1: Codes asso iated with bipartite graphs were onsidered in [2℄, [5℄, [9℄, [12℄{[16℄. In parti ular, Sipser and Spielman [13℄ introdu ed an important idea of estimating properties of a simple iterative de oding pro edure via spe tral properties of the graph G. The main result of [13℄ is given by the following theorem. { 8 { Theorem 4 [13℄ For any " > 0 there exists a polynomial-time onstru tible family of odes with distan e Æ " and rate 1 2H(pÆ) for whi h any < Æ=48 fra tion of errors an be orre ted by a ir uit of size O(N logN) and depth O(logN). The omplexity of a sequential implementation of this de oding is O(N): This theorem gives odes with positive rate R > 0 for 0 < Æ < 0:011: We note that [13℄ used the above family of expander odes with C0 = C1 but with a somewhat di erent de oding algorithm than the ones used in this paper. The present onstru tion together with the de oding pro edure (4) below was used in [15℄ to improve the fra tion of orre table errors in Theorem 4 to Æ=4: 4.3 Repli ated expander odes A modi ation of the above onstru tion was introdu ed in [2℄. Namely, assume that every edge (v; w) 2 E is really a bundle of t parallel edges, ea h with one end in v and the other in w. Further, assume that C0 is a [t ; t R0℄ ode and C1 a [t ; t R1℄ ode. Both odes an be onsidered as binary linear odes, and also as q-ary additive odes (additive subgroups of Fq ; q = 2t). The purpose of introdu ing repli ation is to make use of the remark (1). In parti ular, with C0 = C1 we obtain: Theorem 5 [2℄ For any R0; 0 < R0 < 1; and " > 0 there exists an expander ode C of rate R 2R0 1 and relative distan e Æ = (1 R0)H 1(1 R0) ": Iterative de oding applied to this ode orre ts any < Æ=4 fra tion of errors. This is an improvement over Theorem 4: in parti ular, we obtain odes C with positive rates for all 0 Æ < 0:055: This is the best result known to-date for the fra tion of errors orre table in linear time with expander odes. 5 Attaining apa ity with linear omplexity 5.1 A simple bound Repli ation also enables us to obtain good estimates of the error exponent of iterative deoding. Consider the following de oding algorithm of an expander ode C. For a ve tor y 2 HN 2 let L(y) (R(y)) be the ve tor z su h that for every v 2 A (v 2 B) the ve tor zv is one of the odewords of C0 (C1) losest to yv: Suppose that the transmitted zero ve tor is re eived as y 6= 0: Consider the de oding pro edure of [15℄, [2℄: y(0) = y; y(1) = L(y(0)); y(2) = R(y(1)); y(3) = L(y(2)) : : : (4) We assume that in omputing L(y(0)) the algorithm relies on the representation of C0 as a binary ode. In other words, for every left vertex v the asso iated binary subve tor { 9 { (y(0) v(1); y(0) v(2); : : : ; y(0) v(t )) is de oded with the binary ode C0[t ; t R0℄ into one of the losest odeve tors of C0. All the subsequent stages use the q-ary stru ture of C1 and C0: For instan e, in the se ond stage this amounts to grouping onse utive groups of t bits of the ve tor y(1) into q-ary symbols. More pre isely, for every right vertex w the q-ary subve tor y(1) w = (y(1) w;1; y(1) w;2; : : : ; y(1) w; ) asso iated to it an be written in binary representation as [(y(1) w(1); : : : ; y(1) w(t)); : : : ; (y(1) w((i 1)t+1); : : : ; y(1) w(it)); : : : ; (y(1) w(( 1)t+1); : : : ; y(1) w( t))℄: All the n -subve tors yw are independently de oded with the q-ary ode C1 a ording to the minimum of the q-ary Hamming distan e. The pro edure stops after either having met a xed point (i.e., when y(i+2) = y(i+1) = y(i) for some i) or after having made O(logN) steps. Error exponents of this de oding algorithm for expander odes (repli ated or not) were analyzed in [2℄. The strongest result obtained in that paper is as follows: Theorem 6 [2℄ For a given rate R, any " > 0 and < 1 there exists a polynomial-time onstru tible family of repli ated expander odes of length N su h that Pe(C; p) 2 NF1(R;p); where F1(R; p) = max R R and a ode C1 of rate R1 lose to one. The rst de oding step, y(1) = L(y(0)); removes most of the errors from the re eived word, whi h is possible be ause the ode C0 has relatively large distan e. By taking a large but xed we ensure that, for every left vertex, the error exponent after the rst step is lose the random oding bound E0(R0; p) (Se t. 3.2). At the se ond step, most verti es orrespond to subve tors ontaining less than D1=2 (R0 R)=2 errors and are therefore orre ted. The expanding properties of the graph G ensure that the remaining atypi al subve tors, all them \badly in error", are orre ted at later iterations if their number is small enough. Small enough means smaller than nÆq 1=2 where Æq 1 = D1= is the q-ary relative minimum distan e of C1. Spe i ally, Lemma 3 ensures that if at some point of the de oding pro edure, the number of wrongly de oded subve tors (or verti es of the graph) is less than nÆq 1=2, then the number of subve tors in error must de rease geometri ally at the next iteration (see [2℄ for details). Summarizing, we simply upperbound the probability of a de oding error by the probability that the number of left subve tors remaining in error after the rst iteration is more than nÆq 1=2. 5.2 Re ned te hnique : overview What pre ludes one from obtaining a better error exponent in this way is the fa t that at the se ond step the proof relies on a very strong onvergen e ondition, namely that most right { 10 { verti es re eive fewer than D1=2 errors. Sin e the length of the ode C1 is a xed onstant, we ould in prin iple involve more powerful de oding, but it is un lear how to bound its error rate. Moreover, in orre t odewords obtained in left verti es after the rst iteration tend to be of one and the same Hamming weight (by Proposition 1). This information is also un laimed in the proof of Theorem 6. Our strategy will therefore be to study the probability that the number of verti es in error after the se ond iteration is suÆ iently small for Lemma 3 to apply and again guarantee onvergen e of the de oding algorithm. Our analysis is rst based on the observation that, given the number n of left verti es that are wrongly de oded by the rst iteration, we know that the typi al resulting error ve tor y(1) satis es : 1. for almost every right vertex w, the right subve tor y(1) w has q-ary weight very lose to . This is an appli ation of Lemma 3. 2. almost every edge orresponds to a binary t-tuple whi h is either all-zero or of weight very lose to ÆE(R0)t. This is an appli ation of Proposition 1. Given this information on the right subve tors y(1) w , we want to use a modi ed maximumlikelihood de oder for the ode C1 and estimate the probability that an error will o ur at the se ond de oding iteration. However, the diÆ ulty we fa e is that even if we have a typi al error pattern for almost every y(1) w , we have little ontrol over its probability distribution. To ta kle this problem we will introdu e partial randomization of the overall ode onstru tion and determine an error distribution for almost every right subve tor y(1) w . 5.3 Re ned te hnique : details We modify somewhat the ode family and the de oding algorithm. The odes are the same as in Se tion 4.3, but we will need the full power of Lemma 3, it is important that we hoose bipartite graphs G with the smallest possible , e.g. Ramanujan graphs [11℄ with = O(p ). Furthermore, we introdu e a small amount of randomization: namely, we onsider an ensemble of expander odes obtained by hoosing randomly and independently the oordinate ordering of every vertex sub ode. We take the ode C0 to satisfy the ondition of Prop. 1 and the ode C1 to satisfy Lemma 9 below. The de oding algorithm also onsists of alternating left and right de odings. The rst (left) de oding, L(y(0)); is the same as in (4). The se ond (right) de oding uses a modi ed \max-likelihood" de oding adapted to a non-symmetri additive q-ary hannel, whi h we des ribe in Se tion 5.4. Every subsequent de oding step uses standard de oding for the q-ary symmetri hannel, des ribed in Se t. 5.1. Our rst result, proved in the remainder of Se t. 5, is : Theorem 7 For a given rate R, there exists a polynomial time onstru tible family of repliated expander odes of length N , de ned up to the orderings of the oordinates of the on{ 11 { stituent odes, su h that, given a random set ! of orderings of the onstituent odes, Pe(C; p; !) 2 NF (R;p)(1 "(R;p)); where "(R; p) is a fun tion depending only on R and p su h that "(R; p) < 1 when R < C and "(R; p)! 0 when R! C and where F (R; p) = max R 0, { 12 { Pe X s D1=2[P0("; ; s) + P1("; ; s)℄ (6) where P1("; ; s) = Pr hjSj = s; jFj > "s i P0("; ; s) = Pr[As;"℄ Pr hjT j Æq 0 2 n As;"i; and where As;" stands for the event As;" = fjSj = s; jFj "s g: The rest of the proof is upperbounding Pe. We rst obtain a rough estimate of P1. Denote = s=n. An edge may be \heavy" for two reasons. It may be in ident to the set SF S made up of those verti es for whi h left de oding of the rst iteration outputs a (wrong) odeword of C0 of binary weight at least (1 + ) 0t . Being in ident to SF is a rare event but given that, a high proportion of its edges are likely to be heavy. Or, it may be in ident to S n SF but then being heavy is in itself a rare event. We write P1 P2 + P3; where P2 = Pr hjSF j > " n 2 jSj = ni P3 = Pr hjFj " n jSF j " n 2 ; jSj = ni: Lemma 8 We have P2 Pr[jSj = s℄2 f2(Æ0; )" N and P3 Pr[jSj = s℄2 f3(Æ0; )" N where f2(Æ0; ) and f3(Æ0; ); both positive, stay bounded away from zero as R! C: Proof. We have P2 Pr[jSj = s℄ n " n=2 Pr hj 0 jyj t j 0i" n=2: By Proposition 1 we an write log Pr[ ℄" n=2 ( 0)" N=2. Compared to this, the binomial oeÆ ient n " n=2 an be made to have a negligeable exponent by hoosing t large enough. This gives an exponent f2(Æ0; ) ( 0)=2: Let us now evaluate P3. Take any vertex v of S n SF and let be the proportion of bits in error of the orresponding output odeword of the ode C0. By de nition of SF we have (1 + ) 0. Consider any multiple edge: the random grouping of the t oordinate positions of C0 into multiple edges will give t t possible hoi es for the t edges that will form the multiple edge. The probability that the multiple edge ontains more than (1+ ) t bits in error is therefore = X t k>(1+ ) t t k t t t k t t : (7) { 13 { The dominating term in this last sum o urs for the smallest k; after rearranging and negle ting terms non-exponential in t it is fairly routine to obtain that (1+ ) t(1 )(1 (1+ ) )t t (1 + ) t whi h gives, for small , e 2(1 ) 2t; so that for xed and we obtain that is exponentially small in t in a way that does not depend on . Next observe that if we ompute the probability 0 that a given multiple edge has weight larger than (1 + ) , given that i other given edges have weight larger than (1 + ) , then we will obtain the formula (7) with repla ed by 0 = i and repla ed by 0 < . We will therefore have 0 . With this in mind we write that P3 Pr[jSj = s℄ n " n =2 " n =2: Again, by hoosing t large enough, we obtain that n " n =2 has negligeable exponent and obtain the desired upper bound for P3 with f3(Æ0; ) 1 4 ln 2 0 1 0 2 for small . We see that the exponents of P2 and P3, however small, stay larger than a onstant times N as R ! C: It will therefore be apparent that when R ! C; the probabilities P2 and P3 and hen e P1 are negligeable ompared to P0. Next, we upperbound P0 in (6) by writing: P0 Pr[jSj = n℄ Pr hjT j Æq 0 2 n jSj = n; jSF j " ni: Assume that P0 = 2 NE(R): Sin e we are only interested in exponential behavior of the sum in (6), we simply need to nd the term P0 with the smallest exponent. Therefore, letting Pr hjT j Æq 0 2 n As;"i = 2 NET and swit hing to exponents, we obtain E(R) min Æq 1=2(E0(R0; p) + ET ): (8) Next we evaluate the exponent ET : this is were the random hoi e of oordinate orderings omes in. Be ause the oordinate orderings of the right sub odes have been hosen randomly { 14 { and are independent of the error ve tor (they might very well have been hosen after the rst de oding step), we an argue that ea h right sub ode is, independently of the others, submitted to an error ve tor hosen equiprobably among a set of error ve tors with a given weight pattern. We therefore need to estimate the typi al weight pattern of the error ve tor on a right vertex after the rst de oding step. By Lemma 3, all but a negligeable fra tion of right verti es have more that (1 + ) edges in ident to S, (re all that in Lemma 3 an be made as small as we want be ause = O(p ) and an be taken to be arbitrarily large). This means that almost every right vertex w has an error ve tor of weight at most (1 + ) . Let us establish a further property of these error ve tors. Let Q0 be the subset of Q de ned by the set of those q-ary symbols whose binary representation is of weight at most (1 + )2 0t. We now argue that almost all of subve tors y(1) w have most of their symbols in the subset Q0: Indeed, F is exa tly the set of symbols that do not belong to Q0 after the rst de oding iteration. By Lemma 8 we an assume that jFj " n be ause the opposite event o urs with an exponent that would be mu h greater that the others (and thus negligeable probability) when R! C. By the Markov inequality, the number of right verti es w that have more than in ident edges that orrespond to a symbol in QnQ0 is less than "n= . Now we an simultaneously hoose and " su h that , ", and = "= are all small. Summarizing, we obtain that almost all verti es of B, (i.e. jBj(1 ) of them) have an error ve tor of weight at most (1 + ) and su h that all but of its nonzero symbols belong to Q0, where is again arbitrarily small. We now laim that ET (Æq 0=4 )E1; (9) where E1 is an error exponent for the right de oder, given that the error ve tor has the above pattern. To see this, assume the worst, namely that all jBj verti es with their error ve tor of the wrong pattern will be wrongly de oded. The laim now onsists of saying that if not more than nÆq 0=4 right verti es are in error then the subsequent de oding steps must onverge orre tly. This in turn follows from Lemma 3 whi h implies that the number of left verti es that have more than Æq 0=2 edges in ident to T (the only ones that an be wrongly de oded at the third iteration) an, by hoosing = small enough, be made suÆ iently small (smaller than nÆq 1=4 for example) so that the number of verti es in error will shrink geometri ally at ea h iteration as in [15, 2℄ or se tion 5.1. The hoi e of the fra tion 1=4 is arbitrary and an be repla ed by any number less than 1=2. We next evaluate E1. 5.4 De oding C1 The right de oder assumes that the q-ary error ve tor has weight not more than 0 with 0 = (1+ ), and that among its nonzero symbols, not more than do not belong to the subset Q0. If it does not nd any odeword that ts this hypothesis it returns an arbitrary { 15 { odeword (say, a random one). We have jQj = q = 2t and jQ0j = X k t t k 2H( )t where = (1 + )2 0. Lemma 9 There exists a [ ; R1 ℄ linear q-ary ode C1 su h that for suÆ iently large and t and any > 0 E1 Æq 1 0H( ) 2 : Proof. A nonzero ve tor falls in a random q-ary linear [ ; R1 ℄ ode C1 with probability q (q R1 1): Hen e the average number of odewords of weight i > 0 in C1 with at most nonzero symbols in Q nQ0 equals EAi; q (1 R1) i Xj=0 i j jQ0ji j(q 1)j: When t is large, by (1) we have 1 R1 Æq 1, and the exponents of both binomial oeÆ ients are small. Therefore the above inequality an be rewritten as EAi; . q ( Æq 1+H( )i+ ): We now ompute the error probability for the right de oder under the ondition that the input ve tor is a random ve tor of the required pattern. It an be bounded above by the probability that su h a ve tor overs at least half the symbols of a given ve tor of weight i > 0 and of the above pattern, whi h is not more than q ( H( )i=2+ +"( )); where "( ) > 0 an be made smaller than any given number by an appropriate hoi e of : The probability that the random error ve tor overs half a odeword of weight i is not more than Ai; q ( H( )i=2+ +"( )): (10) As usual, we an hoose a ode C1 su h that every Ai; is not more than EAi; times a polynomial in n. We obtain therefore that the maximum of (10) is obtained when i is as large as possible, namely i = 2 0 . Swit hing to exponents we obtain E1 Æq 1 0H( ) 2 : { 16 { As R ! C; the rst term in (8) tends to zero while the se ond remains bounded away from zero. Together with (9) this enables us to laim that the lower bound (8) is minimized for ! Æq 1 H( 0) = 1 R1 H( 0) = R0 R H( 0) : (11) Substituting this value of in (8), we obtain the exponent F (R; p) of the theorem. 6 A further improvement of the exponent. Borrowing from turboodes In this se tion we show how to modify slightly the ode onstru tion to improve Theorem 7 to Theorem 10 For a given rate R, there exists a polynomial time family of (repli ated, generalized) expander odes of length N , de ned up to the orderings of the oordinates of the onstituent odes, su h that, given a random set ! of orderings of the onstituent odes, Pe(C; p; !) 2 NF (R;p)(1 "(R;p)); where "(R; p) is a fun tion depending only on R and p su h that "(R; p) < 1 when R < 1 H(p) and "(R; p)! 0 when R! 1 H(p) and where F (R; p) = max R EF(R), i.e., expander odes have an exponentially smaller error rate than (quadrati -time-de odable) Forney's on atenated odes. A ommon point that these two families share is the use of two onstituent odes; however in the expander ode onstru tion both odes are binary, while in the on atenated s heme the q-ary ode C1 has a strong algebrai stru ture. We note that in the study of minimum distan e of odes there is a substantial di eren e between expli it families of odes and randomized ensembles. This di eren e does not play su h a prominent role when we study de oding error exponents. Indeed, in this ontext the fo us is on de oding performan e and the goal is to estimate the error probability of some expli it de oding algorithm. It hardly matters whether this probability is omputed for a xed ode and a random error ve tor, or the produ t of a random ode and a random error ve tor. Comparing Theorem 10 with the error exponent of multilevel on atenations (3), we noti e that Ei(R) > F (R; p) starting from some nite (not too large) value of i that depends on the hannel rossover probability p: The following resear h problem suggests itself rather naturally: is it possible to improve the error exponents of expander odes by using several onstituent odes similarly to the improvement (3) by Blokh and Zyablov of Forney's exponent EF(R) ? More generally, now that on atenated odes nally have a rival, it should be very interesting to see whether expander odes (or some enlarged family) ultimately have the potential to provide the best onstru tive exponents. An even more general question is whether there exist polynomial-time de odable ode families with error exponent E(R) su h that the quotient E0(R; p)=E(R) stays bounded as R! C. Finally, we observe that the study of exponents provides an interesting theoreti al framework for trying to dis ern whi h properties of odes are relevant for iterative de oding. Note that riteria su h as minimum distan e ould not have lead to the onstru tion of Se tion 6. Appendix: Proof of Prop. 1 We will study properties of an ensemble A of binary linear odes de ned by (N K) N parityhe k matri es whose elements are hosen independently with P (0) = P (1) = 1=2: Let K = RN: We write f(N) = g(N) if limN!1 1 N log f(N) g(N) = 0: Let C 2 A be a linear ode and Aw be the number of ve tors of weight w in it. It is easy to see that EAw = Nw (2K 1)2 N : { 19 { By the Markov inequality there exist odes with Aw n Nw 2K N (w = 1; 2; : : : ; N): (12) We ontinue with a te hni al lemma. Lemma 11 Let Sj = fx 2 HN 2 : jxj = jg; r w N=2; M(w; r) := X 2Sw fy 2 Sr : d(y; ) rg : Then for N !1; r = N; M(w; r) . nr 2 with equality M(w; r) = nr 2 only for w N 2 (1 ): Proof. M(w; r) = Nw r X i=w=2 wi N w r i : The un onstrained maximum of the summation term is attained when i = wr=N < w=2: Hen e we write M(w; r) = Nw w w=2 N w r w=2 = Nr N r w=2 r w=2 (rewriting the multinomial oeÆ ient) . Nr N r r(1 ) r (N r) : Finding the maximum on w in the last step alls for some explanation. For a xed ve tor y of weight r we ount the number of weight-w ve tors with d(y; ) = r = N: This number is maximized if is a typi al ve tor obtained after n independent drawings from the binomial probability distribution given by P [y+ = 1℄ = ; P [y+ = 0℄ = 1 : Hen e the maximizing argument is given by w=2 = r(1 ): Substituting it, we obtain M(r(1 ); r) = exp[N(H( ) + (1 )H(1 ) + H(1 ))℄ = exp(2NH( )) = Nr 2: { 20 { Now let us ompute the error exponent of maximum likelihood de oding for a ode C of rate R with weight distribution as in (12). Suppose that y 2 HN is a ve tor re eived from the hannel BSC(p) and that the transmitted ve tor is the all-zero one. Let d = ÆGV(R)N: The probability of de oding error an be written as Pe 1 + 2; (13) where 1 = 2d Xw=d d X r=d=2Aw Pr[Ew jyj = r℄; 2 = Pr[jyj d℄; d is the distan e of C and where Ew denotes the event that y is de oded in orre tly to a given odeword of weight w. We have Pr[Ew jyj = r℄ = Nr 1 w X i=w=2 wi N w r i : As above, the sum on w is dominated by the rst term, and we have Pr[Ew jyj = r℄ = w w=2 N w r w=2 Nr : Hen e 1 = 2 n(1 R) max d=2 r d max d w 2d Nw w w=2 N w r w=2 pr(1 p)N r: Let w = !n; r = n. By Lemma 11 the un onstrained maximum on w = !N is attained for ! = 2 (1 ), and then 1 = 2 N(1 R) max d=2 r d nr 2pr(1 p)N r = max ÆGV(R)=2 ÆGV(R) exp[ N(D( kp) + (1 R) H( )℄: The un onstrained maximum on on the right-hand side (the minimum of the exponent) is attained for = 0 (2). We are interested in the ase of R R rit; i.e., 0 ÆGV(R). Then both 1 and 2 behave (in the = sense) as exp( ND(ÆGV(R)kp)): Moreover, the exponent D(ÆGV(R)kp) is attained for = ÆGV(R) and hen e for ! = ÆE(R): To prove that ÆE(R) is indeed the typi al relative weight of in orre tly de oded odewords it is now enough to argue that Pe = 1, i.e., that the estimate (13) is in fa t an asymptoti equality. This is the soalled sphere-pa king bound [10, p. 164℄ whi h states that for any ode with the same rate R we must have Pe 2. { 21 { Thus, for R rit < R < C the error ve tors that lead to a de oding error will be of weight N(ÆGV(R) "1) in all ases of de oding error ex ept a 2 g("1)N fra tion of them (here g( ) is some positive fun tion, not ne essarily the same in di erent expressions). Moreover, the weight of in orre tly de oded odewords will be equal to N(ÆE(R) "2) in all ases of de oding error ex ept a 2 g("2)N fra tion of them. Sin e all error ve tors of any given weight are equiprobable we an now nish the proof by a law-of-large-numbers-type argument. Referen es [1℄ A. Barg, \Complexity issues in oding theory," in Handbook of Coding Theory (V. Pless and W.C. Hu man, Eds.), Vol.I, Amsterdam: Elsevier S ien e (1998), pp. 649{754. [2℄ A. Barg and G. Z emor, \Error exponents of expander odes," IEEE Trans. Inform. Theory 48, no.6 (2002), pp. 1725{1729. [3℄ C. Berrou, A. Glavieux, and P. Thitimajshima, \Near Shannon limit errororre ting oding and de oding: Turboodes", in Pro . of Int. Com. Conf. (ICC'93), pp. 1064{ 1070, 1993. [4℄ E. L. Blokh and V. V. Zyablov, Linear Con atenated Codes, Mos ow: Nauka Publishers (1982), in Russian. [5℄ L. De reusefond and G. Z emor, \On the errororre ting apabilities of y le odes of graphs," Combinatori s, Prob. and Comp., vol. 6 (1997), 27{38. [6℄ I. Dumer, \Con atenated odes and their multilevel generalizations," in Handbook of Coding Theory (V. Pless and W.C. Hu man, Eds.), Vol.II, Amsterdam: Elsevier S ien e (1998), pp. 1911-1988. [7℄ P. Elias, Coding for noisy hannels, IRE Conv. Re ., Mar. 1955, pp. 37{46. Reprinted in D. Slepian, Ed., Key papers in the development of information theory, IEEE Press, 1974, pp. 102{111. [8℄ G. D. Forney, Jr., Con atenated Codes, Cambridge, MA: MIT Press (1966). [9℄ R. G. Gallager, Low-Density Parity-Che k Codes. Cambridge, MA: MIT Press (1963). [10℄ R. G. Gallager, Information Theory and Reliable Communi ation, New York: J. Wiley & Sons (1968). [11℄ A. Lubotsky, R. Philips and P. Sarnak, \Ramanujan graphs," Combinatori a, 8 (3) (1988) 261{277. [12℄ G.A. Margulis, \Expli it onstru tions of graphs without short y les and low density odes," Combinatori a, 2 (1) (1982), 71{78. { 22 { [13℄ M. Sipser and D. A. Spielman, \Expander odes", IEEE Trans. on Inf. Theory, IT-42, no 6, pp. 1710{1722, 1996. [14℄ M. Tanner, \A re ursive approa h to lowomplexity odes," IEEE Trans. Inform. Theory, IT-27, no 5, pp.533{547, 1981. [15℄ G. Z emor, \On expander odes" IEEE Trans. Inform. Theory, IT-47, pp. 835{837, 2001. [16℄ V.V. Zyablov and M. S. Pinsker, \Estimation of the errororre ting omplexity of Gallager low-density odes," Problems Inform. Transm., 11 (1) (1975), 26{36 (in Russian) and 18{28 (English Translation).